The Cost of Adding Parameters to a Model

For a general regression model with n independent observations we consider the variance of the estimate of a quantity of interest under two scenarios. One scenario is where all the parameters are estimated from the data, the other scenario is where a subset of the parameters are assumed known at their true values and the remaining parameters are estimated. We focus on quantities of interest which are deened on the scale of the response variable. We show that, under certain conditions, the ratio of a weighted sum across the design points of the variance of the quantity of interest is given by q=p, where q and p are the number of free parameters in the two scenarios. Thus, in this average sense, the innation in variance associated with adding parameters, also interpreted as the cost of adding parameters to a model, is directly proportional to the number of parameters. We study models involving power transformations, non-linear models and exponential family models.